This invention relates to image acquisition systems for object location and inspection and, more particularly, to techniques for calibrating the optical system in an image acquisition system.
In many applications, it is necessary to determine the two-dimensional location or translation and angular orientation of an object of interest. Depending on the application, a determination of one or more of these properties is called determining the xe2x80x9calignmentxe2x80x9d of an object. In some applications, alignment also may include additional degrees of freedom, such as scale, aspect, shear or even various types of non-linear deformations. For example, in a robotic assembly system, it may be necessary to determine the alignment of a printed circuit board so that a robotic arm can place an integrated circuit onto the board at a precise location. One way to perform this alignment is to mechanically constrain the board at a predetermined location and angular orientation. Guides, brackets and many other well-known mechanical arrangements can be used to accomplish this alignment. However, in other applications, it is not feasible to mechanically constrain each object. In these latter systems, machine vision systems are often used to determine the alignment of objects.
Machine vision systems acquire images of an environment, process the images to detect objects and features in the images and then analyze the processed images to determine characteristics of objects and other features detected in the images. The system generally includes a camera/frame grabber system that generates an image that consists of a plurality of digitized image pixels. The image pixels are then processed with an algorithm implemented in software and/or hardware typically called a vision xe2x80x9ctool.xe2x80x9d
In order for the machine vision system to operate with other equipment, such as a materials handing system or a robotic assembly arm, the output of the machine vision system must converted into a frame of reference that can be used to control the materials handing system. For example, the output of the machine vision system is typically expressed in pixel dimensions, since the underlying image is composed of pixels. The output in pixels must be converted into other metrics, such as millimeters or inches, which can be used to control the materials handing system. Further, it is necessary to convert the output of the machine vision system so that a given point in the image corresponds to a known point in the frame of reference in which the handling system is located and to which the handing system can move.
Conventionally, this conversion is performed in connection with a process called xe2x80x9ccalibration.xe2x80x9d In order to calibrate a machine vision system, a special calibration target, or calibration plate, is mechanically fixed in a known position with respect to the frame of reference. The calibration target includes objects that are located at predetermined points and spaced apart at known distances. In addition, a unique xe2x80x9cfiducialxe2x80x9d mark may be included in order to determine absolute position. The machine vision system then acquires images of the calibration plate objects and the fiducial mark. Since the spacing of the objects is known, the output of the machine vision system in pixels can be directly converted into the necessary metrics for controlling the materials handing portion of the system. The position of the fiducial mark, as determined by the machine vision system, can be used to determine the position of the calibration plate with respect to the frame of reference.
The calibration operation not only permits the necessary conversions, but also can be used to compensate for various distortions introduced by the optical path of the machine vision system. For example, the image acquisition camera may be mounted so that the resulting image is rotated with respect to the frame of reference. Distortions in size, shape, position and aspect ratio of objects may be introduced by the other components of the image acquisition optical system. Since the spacing of the calibration objects are known and the position of the fiducial mark is known, during the calibration process the output of the image acquisition system can be adjusted accordingly in order to remove these distortions.
The manner of determining the alignment of the calibration objects depends on the type of vision tool used to locate the objects. Some tools can tolerate variations in size, shape and angular orientation during the location procedure, but have other deficiencies. For example, the earliest vision tool widely used for object location and inspection in industry was blob analysis. In this type of tool, image pixels are classified as object or background pixels by some means, the object pixels are joined to make discrete sub-objects using neighborhood connectivity rules, and various moments of the connected sub-objects are computed to determine object position, size, and orientation. Blob analysis tools can tolerate and measure variations in orientation and size.
However, such tools cannot tolerate the presence of various forms of image degradation. A more serious problem was that the only generally reliable method ever found for separating object pixels from background pixels was to arrange for the objects to be entirely brighter or entirely darker than the background. This requirement is difficult to achieve in other than the most controlled conditions, although it typically can be achieved if a specially manufactured calibration plate is used.
However, it is very desirable to eliminate the need for a special calibration plate and thus to overcome the limitations of blob analysis tools. In order to overcome such limitations, techniques called xe2x80x9ctemplate matchingxe2x80x9d tools were developed to locate objects based on their pattern rather than grayscale intensities. A template matching tool typically starts with a training step. In this step, a software representation called a pattern, or template, of an image or synthetic description of an ideal object is created and stored. At run-time, the template is moved in various positions over the digitized image and compared to like-sized pixel subsets of the image. The position where the best match between the template and image sub-set pixels occurs is taken to be the position of the object. Because a xe2x80x9csearchxe2x80x9d is performed for the best match, this type of tool is often called a search tool. The degree of match (a numerical value) can be used for inspection, as can comparisons of individual pixels between the template and image at the position of best match.
The first template matching tools used a brightness threshold to reduce the pixels of the template and image to two states: xe2x80x9cbrightxe2x80x9d and xe2x80x9cdark.xe2x80x9d This reduced the computation necessary for the comparison operation to a reasonable level for the available computation facilities. Unfortunately, the thresholding step made sub-pixel accuracy impractical and made the results highly susceptible to the selection of the threshold and variations in illumination and object reflectivity.
Later tools overcame the thresholding problem by using a normalized correlation operation for the template and image comparison step, albeit at the cost of considerable additional computation. Normalized correlation template matching tools also overcame many of the limitations of blob analysis toolsxe2x80x94they can tolerate touching or overlapping objects, they perform well in the presence of various forms of image degradation, and the normalized correlation match value is useful in some inspection applications. Most significantly, perhaps, in order for the tool to operate properly, objects need not be separated from background by brightness, enabling a much wider range of applications.
Unfortunately, while normalized correlation template matching tools work well in determining the location of objects that are translated, they will tolerate only small variations in angular orientation and size: typically a few degrees and a few percent (depending on the specific template). Even within this small range of orientation and size variation, the accuracy of the results falls off rapidly when the objects deviate from fixed sizes and orientations and such a system may not be able to distinguish between two objects that do not differ much in size.
Other search tools have been devised that can determine the alignment of objects in the presence of variations in one or more other characteristics or xe2x80x9cdegrees of freedomxe2x80x9d in addition to translation. For example, such tools may be able to determine the alignment of objects in the presence of significant variations in translation and rotation, translation and size or translation and skew. Other tools can determine the alignment of objects in the presence of variations in multiple degrees of freedom, such as variations in translation, rotation and size. These tools can be used to locate objects in the presence of such variations.
For example, normalized correlation matching tools have been extended to tolerate variations in several degrees of freedom by using-multiple templates in the search. In order to accommodate objects that vary in size, a separate template for each object size is created and stored during the training process. Similarly, in order to accommodate objects that have been rotated, a separate template is created for each object that is rotated with a different degree of rotation.
However, with such tools, a separate pattern must be trained for each object size and each orientation of each object size, and then a search must be performed using each pattern over the entire runtime image. Each pattern occupies computer memory, and, depending on the techniques and the size or complexity of the training image or description, a typical pattern may occupy tens or hundreds of kilobytes or even several megabytes of memory. Aside from the training time required to create the patterns, a large number of patterns can easily consume large amounts of memory. If the patterns are stored in a central location and downloaded to local computers when needed, the network connecting the computers must have significant bandwidth. Further, the need to search the image with a large number of patterns increases the search time significantly. In addition, because patterns must be created for each size and rotation, the patterns are created in discrete steps and the accuracy of the system suffers.
Still other search tools use geometric feature matching to locate and determine the alignment of objects with variations in several degrees of freedom, such as translation, rotation and size. In these systems, a feature detection algorithm produces a geometric description of object boundaries. This geometric description comprises a set of boundary points that lie along contours separating dissimilar regions in the image. Each boundary point specifies both position and orientation. In this system, the training process uses a training pattern to select features to represent the object. Once the boundary points have been generated, they can be transformed by parameterized mathematical algorithms to produce translated, rotated, scaled, and stretched patterns. The pattern search is thereby reduced to searches over parameter values. An example of such a system is the PatMax(copyright) search tool developed and sold by Cognex Corporation, One Vision Drive, Natick, Mass.
Other vision tools are available that can locate and determine the alignment of objects with variations in several degrees of freedom. These vision tools include the HexSight(copyright) Locator machine vision system developed and sold by Adept Technology, Inc., San Jose, Calif. and SMART Search(copyright) machine vision system developed and sold by Imaging Technology, Inc., 55 Middlesex Turnpike, Bedford, Mass. 01730.
With proper vision tools, the calibration operation can be used to eliminate or reduce the effects of camera rotation and optical distortion in an image acquisition system. However, the process is time-consuming and cumbersome. The calibration plate must be carefully fixed in position on the materials conveyor or motion table and a special calibration process initiated. Such a calibration process may need to be repeated if the image acquisition system changes over time. In most cases, the processing of actual materials must be halted during the calibration process.
Therefore, there is a need for a method and apparatus for quickly and easily calibrating an image acquisition system.
In accordance with the principles of the present invention, a fiducial mark with at least one characteristic of known value is placed on one or more of the objects that are examined by the machine vision system. The image acquisition system can be calibrated by examining the objects with a vision tool that can align objects with translation and at least one additional degree of freedom and that can measure the value of the characteristic. In general, information measured by the vision tool can be used to infer similar information concerning the calibration. For example, in one embodiment, the single fiducial mark has a precise dimension and the vision tool can measure the dimension in order to calibrate the scale of the system.
In another embodiment, the object bearing the mark is held in a predetermined position in a mechanical fixture and the vision tool can be used to locate the mark. The outputs of the vision tool can then be used for both size conversion and position with respect to the frame of reference. In addition, if the vision tool can accurately align angular orientation, the outputs of the vision tool can be used to correct for rotation introduced by the optical system. Since the fiducial marks are placed on the objects that are being handled by the system, no special calibration plates or procedures are necessary.
In another embodiment, a normalized correlation search tool is used to locate the mark. The tool internally generates a set of rotated and scaled patterns that are then stored. The stored patterns are subsequently used to perform the search.
In still another embodiment, a normalized correlation search tool internally generates a set of rotated and scaled patterns at runtime when the search for the mark is being performed.
In yet another embodiment, each object to be examined is held in a predetermined position by a fixture on the materials handling system and a fiducial mark is placed on each object. The image acquisition may be calibrated by examining each object, examining objects periodically or examining objects in another pattern, such as randomly. Even if the objects are not fixtured, some recalibration (such as for pixel size alone) may be performed.
In still another embodiment, selected objects to be examined are held in a predetermined position on the materials handling system and a fiducial mark is placed on each object or only on the selected objects. The system may be calibrated by examining the selected objects.